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Executive Summary 



Improving adolescent literacy is a critical step toward improving adolescent academic 
achievement (Kamil , Borman, Dole, Krai, Salinger, & Torgesen, 2008). “Adolescent literacy” 
commonly refers to the skills that students in Grades 4-12 need in order to successfully learn by 
reading, as opposed to learning how to read, which is emphasized in earlier grades (Kamil, 2003; 
Kamil et al., 2008; National Governors Association, 2005). Recent policy reports emphasize the 
need to build students’ reading vocabulary and comprehension skills to meet the increased 
literacy demands that begin in Grade 4 (Carnegie Council on Advancing Adolescent Literacy, 
2010; Meltzer, Smith, & Clark, 2001). Experts who drafted the Common Core State Standards 
for English Language Arts have emphasized that students must show a steadily increasing ability 
to discern more from text to become successful readers (National Governors Association & 
Council of Chief State School Officers, 2010). The current study evaluates an intervention 
{Thinking Reader®) designed to improve middle school students’ reading vocabulary and 
comprehension (Tom Snyder Productions, 2006a). It responds to an interest expressed by 
stakeholders to the Regional Educational Laboratory Northeast and Islands in improving literacy 
outcomes for students beyond elementary school. 

Thinking Reader is a software program for students in Grades 5-8 that incorporates elements 
commonly identified in policy reports as being key components of effective adolescent literacy 
instruction. These reports prioritize elements such as instruction in comprehension strategies 
(Biancarosa & Snow, 2004; Carnegie Council on Advancing Adolescent Literacy, 2010; Meltzer 
et al., 2001); attention to motivation and self-directed learning (Biancarosa & Snow, 2004; 
Meltzer et al., 2001; National Council of Teachers of English, 2006); ongoing formative 
assessment (Biancarosa & Snow, 2004; Carnegie Council on Advancing Adolescent Literacy, 
2010; National Council of Teachers of English, 2006); and inclusion of technology as an 
instructional tool (Biancarosa & Snow, 2004; Kamil, 2003; National Council of Teachers of 
English, 2006). 

When using Thinking Reader, students read novels on computers and respond to prompts. The 
software aims to teach students to use comprehension strategies through a reciprocal teaching 
approach in which the strategies are taught while the teacher and students explore the meaning of 
text (Palincsar & Brown, 1984). Students’ progress is assessed regularly. Thinking Reader also 
aims to motivate students to read and to make self-directed use of strategies. The software has a 
limited but positive evidence base, including statistically significant findings of one quasi- 
experimental study (Dalton, Pisha, Eagleton, Coyne, & Deysher, 2002) and empirical evidence 
supporting instruction on comprehension strategy use (for example, RAND Reading Study 
Group, 2002), particularly on strategy instruction through reciprocal teaching approaches (for 
example, Rosenshine & Meister, 1994). The current study is the first rigorous, randomized 
controlled trial on the program. 

Research Questions and Measures 

This evaluation of the impact of Thinking Reader use by Grade 6 students focused on two 
confirmatory research questions about the effect of the program on two measures of students’ 
reading achievement: 
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1. What is the effect of Thinking Reader on students’ reading vocabulary? 

2. What is the effect of Thinking Reader on students’ reading comprehension? 

A statistically significant impact on either outcome measure would signal the program’s success. 

The study also examined whether Thinking Reader has an effect on two ancillary, but important, 
measures of students’ approaches to reading: 

1. What is the effect of Thinking Reader on students’ use of reading comprehension 
strategies? 

2. What is the effect of Thinking Reader on students’ motivation to read? 

The answers to these questions provide information that may be useful to educators who see 
these factors as being important precursors or supplements to improved achievement. But 
without a direct, measurable effect on reading achievement itself (vocabulary, comprehension), 
such effects would be insufficient to judge the program’s effectiveness. 

The vocabulary and reading comprehension subtests of the Gates-MacGinitie Reading Tests 
(GMRT) (MacGinitie, MacGinitie, Maria, Dreyer, & Hughes, 1999) served as the achievement 
measures for the primary research questions. Two self-report student surveys — the 
Metacognitive Awareness of Reading Strategies Inventory (MARSI; Mokhtari & Reichard, 
2002) and the Motivation for Reading Questionnaire (MRQ; Wigfield & Guthrie, 1997) — served 
as the measures for the ancillary research questions. Each was collected as both a pretest fall 
baseline measure before the start of the intervention and as a posttest spring outcome measure. 

This study also addressed four exploratory research questions. These questions investigate 
whether the impact of the Thinking Reader intervention on students’ reading achievement varied 
across subgroups of students formed on the basis of baseline reading vocabulary, baseline 
reading comprehension, and baseline motivation to read measures: 

1. Does the effect of Thinking Reader on students’ reading vocabulary vary according to 
their baseline reading vocabulary scores? 

2. Does the effect of Thinking Reader on students’ reading comprehension vary according 
to their baseline reading comprehension scores? 

3. Does the effect of Thinking Reader on students’ reading vocabulary vary according to 
their baseline reading motivation scores? 

4. Does the effect of Thinking Reader on students’ reading comprehension vary according 
to their baseline reading motivation scores? 

The outcomes of interest for these exploratory research questions are the vocabulary and 
comprehension subtests of the GMRT that served as the achievement measures for the primary 
research questions. 
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These exploratory research questions were selected because the Thinking Reader program is 
intended to provide differentiated support to students with different skill levels. Specific features 
of the program are intended to help teachers monitor progress and facilitate teachers’ 
individualization of instruction based on students’ demonstrated performance. Exploratory 
Research Questions 1 and 2 are important empirical questions because the literature contains 
very limited evidence to indicate whether an intervention like Thinking Reader might be more or 
less beneficial for students with lower or higher baseline achievement scores. Exploratory 
Research Questions 3 and 4 were selected because, although an interactive software program 
could be motivating for adolescent readers, rigorous evidence does not exist to indicate whether 
an intervention like Thinking Reader might be more or less beneficial for students with lower or 
higher self-reported baseline motivation levels. 

To support the exploratory analyses, we partitioned students’ baseline achievement scores into 
subgroups. The sample was divided into three groups representing the lowest, middle, and 
highest achieving tertiles. Tertiles^ were used instead of the continuous score distribution in 
order to allow the analysis to demonstrate a possible non-linear effect of the treatment across the 
baseline achievement score distribution. The intervention could be ineffective for the majority of 
students, which could drive the results demonstrated for the sample as a whole, but results from 
the tertiles may be able to detect a non-zero effect on a particular, smaller subgroup of students. 

The study collected student, teacher, and school data as covariates for the analyses and collected 
data from classroom observations and electronic report data from Thinking Reader to help 
understand program implementation. Student data included age, gender, and ethnicity, as well as 
English language learner and special education status. Teacher data included information about 
years of teaching experience, educational attainment, and certifications or endorsements held. 
School data included type of school (elementary or middle school), state, enrollment size, and 
the poverty level and ethnicity of students. For a subset of classrooms, structured observations in 
the winter and spring in one intervention and one control classroom at each school provided 
descriptive information about instruction. To examine fidelity of implementation, electronic 
report data were collected from the Thinking Reader program at the end of the year. These data 
provided information about students’ exposure to the software (such as number of books started 
and completed, total number of minutes using the software, and number of weeks spent on each 
book), as well as students’ program levels. 

Description of Thinking Reader 

Students using Thinking Reader read novels on computers and respond to prompts that support a 
range of strategies for understanding text. Users can choose from nine novels with a range of 
difficulty appropriate for middle school readers.^ Development of the program by the Center for 



^ Tertiles were used rather than quartiles or quintiles to avoid having cells with very small sample sizes. The original 
power analysis was not calculated to accommodate dividing the sample into subgroups. 

^ Software licenses for Thinking Reader are available for individual computers, for bundles of 10 licenses, and as an 
unlimited site license. Discounts are given on orders for multiple Thinking Reader novels and for multiple schools 
within a district. As of spring 2010, prices per novel ranged from $250 to $2,200, depending on the type of license 
and number of different novels ordered. Participating schools received unlimited site licenses, at no cost, for three 
Thinking Reader titles during the study year and an additional fourth title for the subsequent year. 
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Applied Special Technology (CAST) began in 2002, and marketing of the program by Tom 
Snyder Productions/Scholastic began in 2004."^ 

The program was designed to embody reciprocal teaching (Brown & Palincsar, 1985; Palincsar, 
1982; Palincsar & Brown, 1984), part of a pedagogical approach for explicitly teaching students 
cognitive strategies to help them understand text. Reciprocal teaching guides students in 
applying four concrete strategies for comprehending text — summarizing, questioning, clarifying, 
and predicting. In summarizing, students explain important information from a piece of text. 
Questioning involves querying a concept in the text to identify key information. Clarifying 
teaches students to question concepts or words that are unclear. Predicting involves using what is 
known from the story to hypothesize what will happen next. 

In addition to these four strategies. Thinking Reader also uses three others — visualizing, feeling, 
and reflecting. Visualizing instructs readers to create mental images of what they are reading. 
Feeling aims to connect students emotionally with what they are reading. Reflecting asks 
students to monitor their progress as readers. Although the last three strategies are not formally 
part of reciprocal teaching, the Thinking Reader program uses the reciprocal teaching approach 
to teach all seven strategies. 

In reciprocal teaching, teachers model specific comprehension strategies and support students’ 
efforts to recall and employ those strategies in their reading. Teachers and students engage in an 
instructional dialogue about text in which teachers initially lead — demonstrating, modeling, and 
providing feedback about strategy use. As students become more competent using the strategies, 
teachers decrease their modeling and students increasingly apply the strategies on their own. In 
Thinking Reader, animated coaches and peers on the computer act as instructors, modeling the 
seven comprehension strategies and prompting students to use them. 

Thinking Reader allows for individualization of instruction. Teachers can customize the amount 
of support to each student by choosing among five levels. The leveling system varies the 
representation of the strategy task, students’ response options, and the availability of the 
animated coaches. Students can progress from responding to highly structured strategy prompts 
(Level 1) to independently selecting their own strategies (Level 5). 

Teachers are encouraged to use Thinking Reader in a three-phase instructional routine. In the 
first phase, teachers introduce the program offline through activities, such as modeling a strategy 
or asking for a summary from the previous session. In the second phase, students read the 
Thinking Reader novel on the computer while the teacher observes, reviews the computer work, 
and conducts conferences with students about the quality of responses. The third phase involves 
additional offline teacher-student interaction, discussing the book or completing an activity to 
illustrate understanding. 



According to Tom Snyder Productions, prior to the start of the study, Thinking Reader was being used at 
approximately 760 schools in 46 states by 67,000 students. These figures are estimates based on the number of 
software licenses sold. 
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Recruitment, Statistical Power, and Study Conditions 

High-need schools were targeted for the study because of the link between high economic need 
and low reading achievement (National Center for Education Statistics, 2009a). Recruitment was 
limited to schools with more than 33% of students eligible for free or reduced-price lunch (high- 
poverty schools). A total of 92 teachers participated from 32 elementary and middle schools in 
16 districts in Connecticut, Massachusetts, and Rhode Island. With a sample size of 2,407 
students and assumptions about the correlation between pretest and posttest scores, the study has 
the statistical power to detect an effect size of 0.19-0.24 standard deviations. 

In each school. Grade 6 reading/English language arts teachers were randomly assigned to 
intervention and control groups after students had been scheduled into classes using the typical 
school procedures. The random assignment produced two groups that did not have statistically 
significant differences in teacher characteristics — including education, certification, and 
experience — or in student achievement measures of reading vocabulary and comprehension and 
self-report measures of comprehension strategy use and motivation for reading. 

Intervention teachers received three Thinking Reader digital novels to read with their students 
during the 2008-09 academic year. Teachers were asked to participate in professional 
development (two, 6-hour group sessions and three individual coaching sessions totaling 7.5-8. 5 
hours) to learn the software and how to use the three-phase instructional routine. They were 
asked to incorporate Thinking Reader into their regular English language arts or reading 
instruction, which was to last 110-165 minutes a week during the time a novel was being 
covered. Each novel was to take 4-6 weeks to complete, with two additional class sessions for a 
culminating activity. Trainers suggested that one novel be covered in the fall, one in the winter, 
and one in the spring — with potential weeks off, as needed or desired, between novels. 

Control teachers used each school’s regular curriculum (business as usual). Students in these 
classrooms engaged in their usual reading/English language arts curriculum and instructional 
program (e.g., reading short stories, newspaper and magazine articles, and novels). 

Analysis and Results 

The final analysis included 90 teachers and a minimum of 2,140 students (89% of the overall 
baseline sample, 90% of the intervention group, and 88% of the control group). A three-level 
model of students nested within teachers nested within schools was used to estimate impact. To 
improve the precision of impact estimates, covariates at Level 1 included students’ pretest 
scores, English language learner status, and special education status; at Level 2 included teacher 
education and years of teaching experience; and at Level 3 included school poverty and school 
size. The analysis also explored whether the intervention effect varied for each outcome or was 
homogeneous across schools. Sensitivity analyses were conducted to test the robustness of the 
results under different scenarios. 

The impact results for the primary research questions indicate that Thinking Reader was no more 
effective than business as usual in improving students’ reading vocabulary (effect size of -0.04) 
or reading comprehension (effect size of 0.03). Results for the ancillary research questions 
indicate that Thinking Reader was also no more effective than business as usual in improving 
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students’ use of reading comprehension strategies (effect size of 0.03) or their motivation to read 
(effect size of -0.03). None of these results are statistically significant. Sensitivity analyses found 
no changes in the direction or magnitude of the intervention effects. 

The data from the classroom observations — conducted twice for each classroom in a subset of 
classrooms — showed statistically significant differences between the intervention and control 
conditions on 47 of 57 measured classroom variables, indicating a contrast in the nature of 
instruction between the intervention and control groups during observations. 

Data from the program’s electronic records showed that students used the Thinking Reader 
program for considerably fewer minutes per week on average — 60 minutes (Book 1), 56 minutes 
(Book 2), and 42 minutes (Book 3) — than the recommended 110-165 minutes. The average 
number of weeks per book — 8.3 (Book 1), 7.1 (Book 2), and 1.7 (Book 3) — also differed from 
the recommendation (4-6 weeks). Book completion fell off from the first to the third book, from 
74% for Book 1, to 53% for Book 2, to 9% for Book 3. Thus the software was not always used as 
prescribed by the developers and modeled in the professional development for intervention 
teachers. 

With regard to the exploratory research questions detailed in Chapter 5, we investigated whether 
the impact of Thinking Reader on student outcomes was different for students with different 
baseline achievement scores or baseline motivation to read, and whether or not impacts for each 
tertiles were statistically different from zero. The multilevel results revealed that the Thinking 
Reader program had no statistically significant effects on any of the subgroups formed from 
baseline scores. In other words, these results confirmed the impact findings. Exploratory findings 
suggest that there is no strong evidence supporting the hypotheses that the Thinking Reader 
program might have differential impact effects on students from different achievement and 
motivation to read subgroups. 

Eleven out of 12 interactions tested across the four exploratory research questions were not 
statistically significant. For the reading comprehension outcome, we found one statistically 
significant interaction (5.77, p =.03). This interaction resulted from the fact that in the lowest 
tertile, intervention students performed 2.15 points higher than control students while in the 
middle tertile, control students outperformed intervention students by 3.61 points. 

However, this statistically significant interaction is difficult to interpret because this finding was 
not replicated on any other contrast or outcome. Due to the large number of post hoc analyses 
involved in the exploratory analyses, this statistically significant interaction may be due to 
chance. Furthermore, when we tested whether the impact effect of the intervention was different 
from zero in each tertile, we found that the Thinking Reader program had no statistically 
significant effects for any of the baseline achievement or motivation to read tertiles. 

Conclusions 

This study was the first randomized controlled trial of Thinking Reader. The study maintained 
the integrity of the randomization throughout. The lack of statistically significant positive effects 
contrasts with the findings of a quasi-experimental study by Dalton et al. (2002). The intent-to- 
treat analytical approach, which analyzes participants on the basis of how they are randomly 
assigned, yielded unbiased estimates of program effectiveness as implemented. Program impact 
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should be interpreted as the effect of being assigned to the intervention condition, not necessarily 
of actually receiving the intervention (Shadish, Cook, & Campbell, 2002). Implementation data 
show that actual take up of the intervention in the way the developer intended was low. The fact 
that dosage and usage varied from the developers’ recommendations reflects the choices that 
practitioners make and the challenges they face when implementing an instructional program. 
The study is limited in that it was not designed to collect more in-depth information on 
implementation. 

The results reported here apply to the implementation of the Thinking Reader software after 
modest professional development, used as a partial substitute for the regular Grade 6 curriculum 
in a whole-group setting — just one of the ways in which the program can be implemented. Use 
of a volunteer sample limits the findings to the schools, teachers, and students that participated in 
the study in Connecticut, Massachusetts, and Rhode Island. 
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Chapter 1. 

Introduction and Study Overview 



Improving adolescent literacy is a critical step in improving adolescent academic achievement 
(Kamil et al., 2008). “Adolescent literacy” commonly refers to the skills that students in Grades 
4-12 need in order to successfully learn by reading, as opposed to learning how to read, which is 
emphasized in earlier grades (Kamil, 2003; Kamil et al., 2008; National Governors Association, 
2005). Recent policy reports emphasize the need to build students’ reading vocabulary and 
comprehension skills to meet the increased literacy demands that begin in Grade 4 (for example, 
Carnegie Council on Advancing Adolescent Literacy, 2010; Meltzer, Smith, & Clark, 2001). 
Experts drafting the Common Core Standards for English Language Arts emphasize that students 
must show a steadily increasing ability to discern more from text to become successful readers 
(National Governors Association & Council of Chief State School Officers, 2010). 

On the 2009 National Assessment of Educational Progress (NAEP), 68% of Grade 8 students 
read below the proficient level — 84% among low-income students (National Center for 
Education Statistics, 2009a). The three states in the Northeast Region that participated in the 
current study (Connecticut, Massachusetts, and Rhode Island) reflect this national problem 
(National Center for Education Statistics, 2009a; Table 1.1). 



Table 1.1 Percentage of Students Testing Below the Proficient Level on the 2009 National Assessment of Educational 
Progress Grade 8 Reading Assessment and Percentage Eligible for Free or Reduced-Price Lunch in Connecticut, 
Massachusetts, Rhode Island, and Nationally 



State 


Below proficient on NAEP — OveralP 


Eligible for free or reduced-price lunch 


Connecticut 


58 


82 


Massachusetts 


57 


81 


Rhode Island 


72 


86 


National 


68 


84 



'‘The number of students assessed (rounded to the nearest hundred by the source) was 2,900 in Connecticut, 4,000 in Massachusetts, 2,800 
in Rhode Island, and 169,100 nationally. NAEP uses weighted sampling to generalize findings to the larger population. 

Source: National Center for Education Statistics (2009a); National Center for Education Statistics (2009b). 



Difficulties with comprehension and self-monitoring of understanding are common among 
middle and high school readers (RAND Reading Study Group, 2002). Because of these 
comprehension problems, many students struggle to learn from advanced texts and may 
disengage from reading (National Governors Association, 2005; RAND Reading Study Group, 
2002). Certain teaching and learning approaches have been shown empirically to improve middle 
and high school students’ reading achievement, including direct teaching of vocabulary and 
teaching a combination of comprehension strategies (Kamil, 2003). The current evaluation 
examines the effectiveness of Thinking Reader®, which uses a software program to directly teach 
vocabulary and evidence-based reading comprehension strategies, supported by specific teacher 
practices (Tom Snyder Productions 2006a, b). 

The evaluation was conducted by Regional Educational Laboratory Northeast and Islands (REL- 
NEI) partners: the American Institutes for Research® (AIR®), the Center for Applied Special 
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Technology (CAST), Tom Snyder Productions®, and Sun Associates. AIR led the evaluation — 
recruiting schools, testing students, and analyzing data. Staff from CAST, a nonprofit research 
and development organization and developer of Thinking Reader, oversaw implementation and 
conducted teacher training along with staff from Tom Snyder Productions, the software 
distributor. To minimize a potential source of bias, the developer and distributor were not 
involved in random assignment, data collection, or data analysis; CAST staff served as 
instructional coaches, and Tom Snyder Productions provided the software and technical support. 
Sun Associates, an education evaluation organization, conducted classroom observations. 

Description of Thinking Reader 

Key features. Thinking Reader is a software program for improving the reading vocabulary and 
comprehension of students in Grades 5-8. CAST began developing the program in 2002, and 
Tom Snyder Productions/Scholastic began marketing it in 2004. With Thinking Reader, students 
read novels on computers and respond to prompts that support seven comprehension strategies 
(explained in detail below). Thinking Reader offers a choice of nine novels with a range of 
difficulty appropriate for middle school readers.^ 

The program embodies an approach to reading instruction known as reciprocal teaching 
(Palincsar, 1982; Palincsar & Brown, 1984; Brown & Palincsar, 1985), which requires teachers 
to model comprehension strategies and support students’ efforts to recall and employ those 
strategies in their reading. In Thinking Reader, animated coaches and peers play the instructional 
role, modeling the seven comprehension strategies and prompting students to use them at 
appropriate points in the text. 

Teachers are expected to play an active role — monitoring and individualizing instruction, 
assessing students’ progress, and customizing the support provided to each student. The program 
has five levels of support, which vary the representation of the strategy prompt, students’ 
response options, and the availability of the animated coaches. Students can progress from 
responding to highly structured strategy prompts (Level 1) to independently selecting strategies 
(Level 5). According to the distributor, when the program is used at lower levels, supports focus 
mostly on helping students understand text using direct hints on to how to answer prompts (e.g., 
sentence starters). Higher levels focus more generally on how to use and apply comprehension 
strategies and include more questions requiring open-ended responses (Tom Snyder Productions, 
2006). Representative screenshots are presented in Appendix A (see Exhibits A.1-A.7). 
According to the distributor, teachers are expected to adjust the level of support to individualize 
instruction for each student based on their progress and development. As students gain more 
skills and confidence as readers, they should become more independent and use the program 
supports less (Tom Snyder Productions, 2006). 

Teachers are encouraged to use Thinking Reader in a three-phase instructional routine based on 
the developer’s guidelines and commonly recommended reading pedagogy of before, during, and 



^ Participating schools received unlimited site licenses for four Thinking Reader titles at no cost. For reference, the 
following price information, as of spring 2010, was provided by the distributor: Prices per novel range from $250 to 
$2,200, depending on the type of license and number of different novels ordered. Licenses for Thinking Reader are 
available for individual computers, for bundles of 10 licenses, and as an unlimited site license. Discounts are given 
on orders for multiple Thinking Reader novels and for multiple schools within a district. 
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after reading activities (Duke & Pearson, 2002). The routine calls for the teacher to introduce the 
program offline through activities, such as modeling a strategy or asking for a summary from the 
previous session. In the second phase, students read the Thinking Reader novel on the computer 
while the teacher observes them, reviews their work, and meets with students about the quality of 
their responses. The third phase involves additional offline teacher-student interaction, 
discussing the book or completing an activity to illustrate understanding. 

Figure 1 . 1 is a logic model that depicts how the Thinking Reader program of instruction — a 
combination of software use and supporting teacher practices — is intended to improve student 
outcomes. Supporting teacher practices include student-teacher conferencing, comprehension 
strategy instruction, and implementing the three-phase instructional routine. The Thinking 
Reader program of instruction is intended to improve two related measures of students’ reading 
achievement: students’ reading vocabulary and comprehension skills. In addition, the program is 
intended to directly increase students’ knowledge and use of seven reading comprehension 
strategies (described in detail below) and increase students’ motivation to read. The two ancillary 
outcomes, use of reading comprehension strategies and motivation to read, may also be 
important precursors or supplements to improved reading outcomes in vocabulary and 
comprehension. 



Figure 1.1 Logic Model for Thinking Reader Software Program 



TAmlrmjgi ReacTerP ro g ram 



r 



Primary Outcomes 







Reading Vocabulary 
Reading Comprehension 



Ancillary Outcomes 

Use of Reading Comprehension Strategies 
Motivation to Read 

V J 



Source: Study team compilation. 



Additional components. Engaged readers have been described as readers who are motivated to 
read and are socially interactive in their approach to reading (Guthrie & Wigfield, 2000). The 
influential policy report Reading Next identified motivation and self-directed learning as being 
important to adolescent reading achievement (Biancarosa & Snow, 2004). Thinking Reader aims 
to address engagement, motivation, and self-directed learning in at least four ways. First, 
scaffolding supports in the program alter the level of challenge for students, providing students 
with choices and control over access to supports and options for their response. Second, self- 
assessment allows students to reflect on their progress and set goals. Third, the program sets up 
students to interact with one another around a text. Thinking Reader is intended to be integrated 
with classroom discussion and peer interaction. Last, the novels in Thinking Reader are selected 
to be age appropriate to middle school students and relevant to their real-life experiences. 
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Thinking Reader has features that can be tailored to match a set of principles called universal 
design for learning. Although this set of principles has not been rigorously tested, it is based on 
the premise that teachers should individualize instruction and use a flexible learning environment 
to meet students’ needs (Rose & Meyer, 2002). In addition to the levels of support. Thinking 
Reader incorporates ways to adjust the text size, text contrast, and screen color; a text-to-speech 
tool so text can be read aloud; a highlighter function to track text as it is read aloud; the option of 
a human recorded narration or a synthetic voice whose speed and pitch can be adjusted; and an 
option for recording rather than typing answers. 

Literature Review 

Evidence base forT\imk\ng Reader. The research relevant to this evaluation includes one 
rigorous study on the Thinking Reader program (Dalton, Pisha, Eagleton, Coyne, & Deysher, 
2002) and studies that examined the effectiveness of the approaches to reading instruction used 
by Thinking Reader (e.g., Deno, 2003; Gambrell & Jawitz, 1993; Palincsar & Brown, 1984; 
Rosenshine, Meister, & Chapman, 1996; What Works Clearinghouse, 2010). A review of this 
body of knowledge showed enough positive (if not always highly rigorous) evidence to warrant 
an evaluation of the Thinking Reader software with a randomized controlled trial design. 

The direct evidence base for Thinking Reader is limited. The current study is the first 
randomized field trial to test its effectiveness. A previous quasi-experimental study evaluated a 
pilot version of the software among a sample of 102 struggling middle school readers (average 
age, 12.5 years). After controlling for pretest scores on reading vocabulary and comprehension, 
the study found that students in the Thinking Reader intervention demonstrated significantly 
greater gains in comprehension than the comparison group on the Gates-MacGinitie Reading 
Tests (GMRT) (MacGinitie, MacGinitie, Maria, Dreyer, & Hughes, 1999; Dalton et al., 2002). 
The effect size was 0.29, approximately half a grade level of reading achievement gain. 

That evaluation differed from the current work in several ways. First, a program of traditional 
(i.e., noncomputer-based) reciprocal teaching methods was delivered to the control group.^ 
Second, because the software was a pilot program, it had fewer features — less scaffolding and 
fewer assessments. Finally, the analysis focused only on struggling readers who had performed 
at the 25th percentile or lower on the pretest. In that study, 80% of the students were in special 
education and 79% of the teachers taught the intervention in a special education class of 12-14 
students (B. Dalton, personal communication, December 7, 2009) — a smaller setting than 83% of 
the classes in the current study. 

Evidence base for instructional features used byTlvivMng Reader. Evaluating Thinking 
Reader in an experimental study is warranted based on the statistically significant findings of one 
quasi-experimental study of the program and its use of strategies and instructional approaches 
with empirical evidence. In addition, the high priority given to improving adolescent literacy by 
stakeholders in the northeast states when the REL-NEI study was in its start-up phase and the 
call for more research on educational technology in general (Moran, Ferdig, Pearson, Wardrop, 

& Blomeyer, 2008) also warranted conducting a randomized controlled trial on Thinking Reader. 



® This was a deliberate decision to make the intervention and control conditions as comparable as possible so that the 
main difference between the groups was whether Thinking Reader was used. 
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Comprehension strategies. Thinking Reader aims to teach comprehension strategies through 
reciprocal teaching (developed by Palincsar 1982; also Brown & Palincsar, 1985; Palincsar & 
Brown, 1984), which involves guided practice in applying four strategies to reading 
comprehension — summarizing, questioning, clarifying, and predicting. In an instructional 
dialogue about text between a teacher and students, the teacher initially takes the lead — 
demonstrating, modeling, and providing feedback on strategy use. As students gain competence 
in using the strategies, the teacher does less modeling and students apply the strategies on their 
own (Au & Raphael, 1998; Duke & Pearson, 2002; Pearson & Gallagher, 1983). With scaffolded 
learning, the teacher adjusts instruction to meet the needs of the student based on the demands of 
the task, gradually releasing responsibility to the student (Vygotsky, 1978; Wood, Bruner, & 
Ross, 1976). As McKenna (1998) suggests, this type of guidance and scaffolding customized to 
the student can be offered by talking book computer software. 

Rigorous studies indicate that reciprocal teaching strategies can show positive results for reading 
(Brown & Palincsar, 1989; Palincsar & Brown, 1984, 1989; Rosenshine & Meister, 1994; What 
Works Clearinghouse, 2010). In two early experiments using groups of students randomly 
assigned to a reciprocal teaching condition or a control condition, Palincsar and Brown (1984) 
reported that students receiving reciprocal teaching strategy instruction showed statistically 
significant positive effects on comprehension scores on daily assessments. Research syntheses 
have shown mixed results for the strategies. In a meta-analysis of experimental or quasi- 
experimental studies examining the effect of reciprocal teaching, Rosenshine and Meister (1994) 
found an average effect size of 0.20 (range: -0.55 to 0.77) using standardized tests of reading 
comprehension as outcome measures. The What Works Clearinghouse (2010) reviewed six 
studies on reciprocal teaching and found mixed results for comprehension outcomes. Across the 
studies, participants included 316 students from Grades 4 to 12 who experienced reciprocal 
teaching interventions at their schools for periods of time ranging from 6 to 30 days of 
intervention. The group of studies included both teacher- and researcher-implemented sessions. 
The What Works Clearinghouse rated reciprocal teaching with improvement indices in 
comprehension from -23 to 42 percentile points, with an average of 6 percentile points. 

Thinking Reader expands on the reciprocal teaching approach by providing guided practice in 
seven reading comprehension strategies (Table 1.2). The first four are the strategies commonly 
used in reciprocal teaching. In summarizing, students explain information gleaned from a piece 
of text. Questioning involves identifying key information about a concept in the text by asking 
questions. Clarifying involves asking about concepts or words that are unclear. Predicting entails 
using what is known from the story to hypothesize what will happen next. The three additional 
strategies are visualizing, feeling, and reflecting. Visualizing involves instructing students to 
create mental images of what they are reading. The feeling strategy aims to connect students 
emotionally with what they are reading. The reflecting strategy involves students in monitoring 
or reflecting on their progress as readers, helping them to become more strategic and efficient in 
their approach to reading. 

The reciprocal teaching literature generally looks at the four strategies applied in combination; 
however, two of the strategies — summarizing and questioning — have been studied individually. 
The National Reading Panel (National Institute of Child Health and Human Development, 2000) 
conducted a comprehensive review of experimental and quasi-experimental reading-related 
studies. Eighteen studies examined summarizing, 72% of them for students in Grades 5 and 6. Of 
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the 18 studies, 1 1 reported positive effects on the quality of written summaries, with improved 
recall and question answering. Only two studies looked at effects on standardized tests. Overall, 
the comprehensive review concluded that the summarization strategy led to improvement in 
students’ memory for text and their ability to identify main ideas. The questioning strategy had 
the strongest body of evidence, with 27 studies, in the National Reading Panel report. Reporting 
on 26 of these studies (conducted in Grade 3 through college) in a meta-analysis of quasi- 
experimental and randomized controlled trial studies, Rosenshine , Meister, and Chapman (1996) 
found that 17 studies used questioning as the single intervention.^ Of those, 7 used a standardized 
outcome measure (mean effect size of 0.25, with a range of -0.25 to 0.70); 1 1 used a research- 
specific outcome measure (mean effect size of 0.79, with a range of 0.00 to 1.37). Thus, 
summarizing and questioning have more evidence of transfer to researcher-generated tests (near 
transfer) than to standardized tests (far transfer). Clarifying and predicting have not been studied 
rigorously in isolation. 

No rigorous studies have examined the feeling or reflecting strategies. However, Rosenblatt 
(1994) theorized, for example, that if students think about how they are feeling about a story, its 
characters, and the plot, they will be more engaged. The National Reading Panel (National 
Institute of Child Health and Human Development, 2000) reported on seven experimental studies 
(for Grades 2-8) that examined visualizing and concluded that it can help students improve their 
memory for the text. Joffe, Cain, and Marie (2007) theorized that making a mental image of what 
is read can help with memory because information is being encoded into two separate mental 
systems: one verbal and one non-verbal. Studies have shown that, after being trained to create 
mental pictures while reading, students in intervention groups outperform those in control groups 
on story-related questions (Gambrell & Jawitz, 1993; Oakhill & Patel, 1991; Pressley, 1976). 



Table 1.2 Key Reading Comprehension Strategies Used in Thinking Reader 



Strategy 


Definition 


Purpose 


General example 


Thinking Reader 
example 


Summarizing 


Students retell the 
main points of what 
they have read. 


Helps students 
remember and 
understand what 
they are reading. 


After students read a 
book, the teacher asks 
them to write a short 
summary. 


The software prompts 
students to select the 
best summary. 


Questioning 


Students pose a 
question about 
something that is 
important to learn 
and remember from 
what they have read. 


Requires students 
to identify key 
information. 


While reading a book 
as a class, students are 
asked to write questions 
about important parts of 
the chapter and then to 
answer the questions. 


The software prompts 
students to select the 
most important question 
from a list and then to 
answer it. 


Clarifying 


Students ask about 
something they do 
not understand in 
what they have read. 


Helps students 
resolve confusion. 


While reading a book 
as a class, students ask 
questions about 
something that is 
confusing. 


The software asks 
students to identify 
something that is 
confusing in the text. 



’ Nine studies used questioning in the context of an intervention that used two or four of the reciprocal teaching 
strategies. 
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Strategy 


Definition 


Purpose 


General example 


Thinking Reader 
example 


Predicting 


Students use what 
they have read to 
predict what will 
happen next in the 
novel. 


Helps students 
analyze text 
content. 


After reading a few 
chapters, students are 
asked to predict what 
will happen next in the 
novel. 


The software asks 
students to make a 
prediction about what is 
going to happen next in 
the text. 


Visualizing 


Students use the 
information they 
have read to create a 
picture in their 
minds. 


Gives students a 
way to remember 
what they read. 


After reading a chapter, 
students are asked to 
create a mental picture 
of a key occurrence in 
the chapter. 


The software asks 
students to look at 
highlighted words in the 
text and visualize what 
is happening. Then, the 
software prompts 
students to write or 
record what they see in 
their minds. 


Feeling 


Students make 
explicit their 
feelings about the 
story, a character, or 
the plot. 


Helps students 
connect with the 
text. 


The teacher asks 
students to describe 
how they feel about a 
particular event in the 
story. 


The software prompts 
students to describe 
their feelings or how 
they would feel if they 
were one of the 
characters. 


Reflecting 


Students track their 
progress as a reader, 
monitoring their 
comprehension. 


Helps students 
decide which 
reading strategies 
work for them and 
why. 


Teachers ask students 
to write self-evaluations 
of their own progress as 
readers in journals. 


The software prompts 
students to describe and 
evaluate their progress. 



Note: The first four strategies are part of the reciprocal instruction model. 

Source: Duke and Pearson (2002); Harris and Hodges (1995); Tom Snyder Productions (2006b). 



Vocabulary instruction. In addition to comprehension strategy instruction, Thinking Reader 
aims to build students’ voeabulary by exposing them to new words in the eontext of authentie 
literature (Tom Snyder Productions, 2006b). The program provides the opportunity for explicit 
instruction when teaehers introduee and discuss vocabulary pertinent to the story. Target words 
are highlighted on the screen. Students can click on these words, which are hyperlinked to a 
glossary. Definitions are also available in Spanish. 

Research suggests that word knowledge is correlated with measures of reading comprehension 
(Pearson, Hiebert, & Kamil, 2007); however, establishing the causal link between increasing 
vocabulary and increasing reading ability has been difficult (Stanovich, 2000). Nevertheless, 
students with good eomprehension skills know many more words than struggling readers and are 
better able to use contextual cues and word structure knowledge to determine word meaning 
(Nagy & Scott, 2004). In their review, Blaehowiez, Fisher, Ogle, and Watts-Taffe (2006) suggest 
that exposure to new and sophisticated words, including ones presented in a read-aloud 
environment, help to build a “word eonsciousness” for students. A meta-analysis of voeabulary 
interventions (conducted at Grade 2 through college) by Stahl and Fairbanks (1986) suggests that 
voeabulary interventions ean have a statistically significant positive effect on reading 
comprehension outcomes (mean effect size of 0.97 for 41 analyses of comprehension measures 
that eontain words targeted in the intervention; mean effect size of 0.30 for 15 analyses of 
standardized tests not designed to contain targeted words). These interventions also had a 
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statistically significant positive effect on measures of vocabulary knowledge (effect size of 0.20 
for 17 analyses using global vocabulary measures). The most effeetive voeabulary interventions 
have been “mixed-methods” that provided both definitions and contextual information for words 
(Blaehowiez , Fisher, Ogle, & Watts-Taffe, 2006). The Thinking program uses this 

approach. 

In addition to support for direct vocabulary instruction in general, the National Reading Panel 
(National Institute of Child Health and Human Development, 2000) concluded that a small 
literature base supports the use of eomputers for vocabulary instruction. For example. Reinking 
and Rickman (1990) tested whether providing Grade 6 students with definitions on a computer 
sereen had an impaet on their vocabulary and comprehension outeomes. Students randomly 
assigned to interventions in which they read passages on a computer and received optional or 
mandatory eomputer voeabulary assistanee outperformed eomparison groups who read exeerpts 
on paper with assistance from either a dictionary or glossary. Statistically significant differences 
were found between the two groups for experimenter-devised voeabulary and comprehension 
tests. Vocabulary assistance from a computer may be particularly helpful as students read longer 
and more difficult material (Greenlee-Moore & Smith, 1996). 

Assessment and feedback. Thinking Reader is designed to support teachers’ assessment of 
students and students’ self-evaluation. Computerized comprehension quizzes provide teachers 
and students with information to monitor students’ understanding. Responses to strategy prompts 
are collected in an electronic worklog for review during student-teaeher eonferenees. Teachers 
can also provide students with electronic feedback and requests for revisions to their strategy 
prompts. Responding to eertain strategy prompts gives students opportunities to refleet on their 
progress as readers. 

The poliey report Reading Next suggests that teaehers should informally assess their students 
regularly to understand how adolescent readers are progressing (Biancarosa & Snow, 2004). 
Ongoing classroom-based assessments are intended to give teachers immediate feedback about 
student performance and to help them adjust the instruction that students are receiving, as needed 
(Torgesen & Miller, 2009; Winograd, Flores-Duenes, & Arrington, 2003). Frequent monitoring 
of students’ progress is intended to prompt teachers to intervene as soon as a learning diffieulty 
is encountered and to help students learn to self-assess and set appropriate goals (Cioffi & 
Carney, 1997; Lipson & Wixson, 2003). Deno’s (2003) review of both experimental and 
correlational studies supports teachers’ use of progress monitoring — associating it with high 
levels of student achievement, improved teaeher deeision making, and increases in students’ 
awareness of their own performance (see also Fuchs & Fuchs, 1999 for an earlier review of 
monitoring student progress). 

Learner control. As Proctor, Dalton, and Grisham (2007) point out, a key issue in designing 
digital texts with embedded learning supports is learner eontrol and whether eomputer support 
should be “pushed” to the student, which help to ensure that features are experienced. The 
alternative is to have students be more self-direeted, allowing support features to be “pulled” by 
the student when he or she judges the support to be needed. This design question centers on 
whether students in a more learner-controlled environment will appropriately aceess supports. In 
their review. Shin, Schallert, and Savenye (1994) describe several studies that show it is only 
students with higher skill and inquiry levels that do better and work effieiently using programs 
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with more learner control. Indeed, some researchers claim that the students who would benefit 
the most from computer support are least likely to access supports appropriately (e.g., Anderson- 
Inman, Homey, Chen, & Lewin, 1994). Struggling readers may find a high number of features 
and options in a program distracting, doing things like accessing definitions for words they 
already know (Boone & Higgins, 1993; Dalton & Strangmann, 2006). In one study, high- and 
low-skilled fifth- and sixth-graders read passages in both printed and computer presentations. 
Students who were not the highest achieving benefitted more from computer-controlled 
environments, perhaps because their metacognitive skills were not as developed (Reinking, 

1988). One option is to give students control of their environment, thus reinforcing self-direction, 
while having the computer advise them on their choices. In a review of studies. Shin et al. (1994) 
found that students in learner-controlled environments with computer guidance receive higher 
scores than student in control conditions. Thinking Reader uses a combined “push-pull” 
approach. Some features — such as text-to-speech, glossary hyperlinks, and hints by animated 
coaches — are accessed only when students choose to use them. Other features, such as 
explanation of and prompts to use comprehension strategies and periodic comprehension checks, 
are built so that students automatically experience them. 

Research Questions 

This evaluation of the impact of Thinking Reader use by Grade 6 students focused on two 
confirmatory research questions about the effect of the program on two measures of students’ 
reading achievement: 

1. What is the effect of Thinking Reader on students’ reading vocabulary? 

2. What is the effect of Thinking Reader on students’ reading comprehension? 

A statistically significant impact on either outcome measure would be a signal of the program’s 
success. 

The study also examined whether Thinking Reader has an effect on two ancillary, but important, 
measures of students’ approaches to reading: 

1. What is the effect of Thinking Reader on students’ use of reading comprehension strategies? 

2. What is the effect of Thinking Reader on students’ motivation to read? 

The answers to these questions provide information that may be useful to educators who consider 
these factors to be important precursors or supplements to improved achievement. But without a 
direct measurable effect on reading achievement (vocabulary, comprehension), such effects 
would be insufficient to judge the program’s effectiveness. The impact results are presented in 
Chapter 4. 

We also specified a set of four exploratory research questions that examined whether the impact 
of the Thinking Reader intervention on students’ reading achievement varied across different 
subgroups of students. The subgroups were formed on the basis of baseline achievement and 
motivation to read measures: 
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1. Does the effect of Thinking Reader on students’ reading vocabulary vary according to their 
baseline reading vocabulary scores? 

2. Does the effect of Thinking Reader on students’ reading comprehension vary according to 
their baseline reading comprehension scores? 

3. Does the effect of Thinking Reader on students’ reading vocabulary vary according to their 
baseline reading motivation scores? 

4. Does the effect of Thinking Reader on students’ reading comprehension vary according to 
their baseline reading motivation scores? 



The outcomes of interest for these exploratory research questions are the vocabulary and 
comprehension subtests of the GMRT that served as the achievement measures for the primary 
research questions. The exploratory results are presented in Chapter 5. 

Study Design 



The study, a multisite cluster randomized controlled trial, randomly assigned Grade 6 teachers to 
intervention or control groups within schools. The study involved reading/English language arts 
teachers in 32 elementary and middle schools in 16 districts in Connecticut, Massachusetts, and 
Rhode Island. The study sample consisted of 92 teachers and 2,407 students. Teachers and their 
intact classes were randomly assigned to one of the two groups after students had been scheduled 
into classes using the typical school procedures. Teachers assigned to the control group were to 
use the regular school curriculum (business as usual). A well-conducted randomized controlled 
trial design that compares an intervention group with a business-as-usual control group generates 
statistically unbiased estimates of the effects of an intervention on the outcomes of interest 
(Shadish, Cook, & Campbell, 2002). 

The primary outcomes of interest — reading vocabulary and comprehension — were measured 
using two subtests of the standardized GMRTs (MacGinitie et al., 1999). Results for two 
additional, ancillary outcomes — students’ use of comprehension strategies and students’ 
motivation to read — were measured using two existing self-report instruments: the 
Metacognitive Awareness of Reading Strategies Inventory (MARSI) (Mokhtari & Reichard, 
2002) and the Motivation for Reading Questionnaire (MRQ) (Wigfield & Guthrie, 1997). Data 
from all measures given at the beginning and end of the year were used to estimate the impact of 
the intervention. Sensitivity analyses were conducted, and usage data from the software were 
analyzed to examine implementation fidelity. An existing instrument was modified for 
conducting structured classroom observations for a subset of classrooms to check fidelity of 
implementation and to explore possible differentiation between intervention and control 
classrooms in classroom instruction. 



Intervention group. Schools received the three Thinking Reader digital novels that they selected 
during the 2008-09 academic year. Intervention teachers were asked to participate in 
professional development (two, 6-hour group sessions and three individual coaching sessions 
totaling 7. 5-8. 5 hours) to learn the Thinking Reader software. Intervention teachers were trained 
to use the three-phase instructional routine. They were asked to incorporate Thinking Reader into 
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their regular English language arts or reading instruction for 110-165 minutes a week while a 
novel was being covered. Teachers were told that each novel should take 4-6 weeks to complete, 
with two additional class sessions for the culminating end-of-novel activity. Trainers also 
suggested that one novel be covered in the fall, one in the winter, and one in the spring — with 
weeks off, as needed or desired, between novels. 

Control group. Classrooms in the control group used their schools’ regular curriculum (business 
as usual). Students in these classrooms engaged in the regular activities of their usual English 
language arts literacy curriculum and instructional program (e.g., reading short stories, 
newspaper and magazine articles, and novels). 

Organization of This Report 

Chapter 2 describes the study methodology. Chapter 3 provides an overview of implementation 
of the Thinking Reader intervention, including data from computer worklogs and classroom 
observations. Chapter 4 describes the results of the impact analyses. Chapter 5 summarizes the 
study’s findings. Appendices provide detailed information about the Thinking Reader program, 
the technical approach to the study, and the analytical decisions made. 
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